Add JSON to Parquet Variant conversion with shredding - #8391
Conversation
860196c to
b9ae199
Compare
Merging this PR will not alter performance
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | slice_empty_vortex |
310 ns | 368.3 ns | -15.84% |
| ⚡ | Simulation | rebuild_naive |
111 µs | 98.9 µs | +12.32% |
| ⚡ | Simulation | bitwise_not_vortex_buffer_mut[128] |
273.6 ns | 244.4 ns | +11.93% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing adamg/variant-json-handling (8318ee1) with develop (71319b7)
3e823bc to
0bc0405
Compare
67592eb to
6b3a236
Compare
6b3a236 to
d78dcaa
Compare
|
Do these functions mix DType (JSON) with encoding (ParquetVariant)? At least the docs seem to suggest that they do. We probably want vortex-json to expose to_json and from_json functions (or maybe this just a cast kernel) and then implement them using execute_parent for ParquetVariant. But it seems like you are actually defining a more general DType::JSON -> DType::Variant (any variant encoding) function? In which case, it cannot be a cast because you want the shredding options, but the definition should probably still live in the vortex-json crate, and implement general-purpose shredding (per the canonical Variant array), without touching the non-shredded data? |
|
@gatesn I've really tried to keep expressions that can't be evaluated directly from the core. I thought execute_parent rules were less flexible but I guess I can try and do that. |
a13a0a2 to
8b284a0
Compare
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
8b284a0 to
d6355b9
Compare
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
d6355b9 to
8cb09a9
Compare
|
I think I addressed the comments. I've also made the PR a bit smaller and more focused, just on |
robert3005
left a comment
There was a problem hiding this comment.
I think the only thing is the implicit JsonToVariant for VarBinView. We should require an extension json type at this level.
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Summary
This PR adds a
vortex.json_to_variantexpression for converting UTF-8 orvortex.jsonextension arrays into Variant arrays.The expression lives in
vortex-jsonso JSON-facing expression metadata and serialization are independent of a concrete Variant encoding.vortex-parquet-variantprovides the concrete execution kernels, parsing JSON throughparquet_variant_computeand producing Parquet Variant storage.